Imaging processing methods and systems using a target 3D model by performing 3D reconstruction on a target image

ABSTRACT

The present disclosure relates to an image processing method and apparatus, an electronic device and a computer readable medium. The image processing method includes: acquiring an image to be processed, the image to be processed including a target portrait; obtaining a target three-dimensional (3D) model by performing a 3D reconstruction operation on the target portrait in the image to be processed; obtaining a portrait lighting image by performing portrait lighting processing on the target 3D model, and extracting a first image including the target portrait from the portrait lighting image; and merging a second image including a photographic background cloth and the first image into a target image, the second image being used as a background image of the first image.

CROSS-REFERENCE TO RELATED APPLICATION

The application claims priority to Chinese patent application No.201810566077.3, filed Jun. 4, 2018, titled “Image Processing Method andDevice, Electronic Device and Computer Readable Medium”, the disclosureof which is incorporated herein by reference in its entirety as part ofthe application.

TECHNICAL FIELD

The present disclosure relates to the technical field of imageprocessing, in particular to an image processing method and apparatus,an electronic device and a computer readable medium.

BACKGROUND

With the rapid development of smart terminal technology, smart terminaldevices, especially smart mobile phones, have entered thousands ofhouseholds. Existing smart mobile phones integrate a variety offeatures, e.g., high-definition photographing function. The portraitphotographing applications of the existing smart mobile phones canusually beautify images taken by the cameras of the smart mobile phones.

However, when users wish to take a formal photo (e.g., a registrationphoto), they still need to go to a professional studio. Due to thephysical limitations of portrait photography of smart mobile phones infacial distortion, scenery, lighting and definition, smart mobile phonescan merely be used for entertainment and are unable to take professionalportrait photos, for example, formal photos taken by professionalstudios.

SUMMARY

At least one embodiment of the present disclosure provides an imageprocessing method, which includes: acquiring an image to be processed,the image to be processed comprising a target portrait; obtaining atarget three-dimensional (3D) model by performing a 3D reconstructionoperation on the target portrait in the image to be processed; obtaininga portrait lighting image by performing portrait lighting processing onthe target 3D model, and extracting a first image comprising the targetportrait from the portrait lighting image; and merging a second imagecomprising a photographic background cloth and the first image into atarget image, the second image being used as a background image of thefirst image.

In the method according to some embodiments of the present disclosure,merging the second image comprising the photographic background clothand the first image into the target image includes: taking the secondimage as an image of a background layer, and taking the first image asan image of a foreground layer; determining in the image of thebackground layer a pixel corresponding to a pixel of the image of theforeground layer, based on position information on the first image inthe image to be processed; and replacing the pixel of the image of thebackground layer with the corresponding pixel of the image of theforeground layer to obtain the target image.

In the method according to some embodiments of the present disclosure,obtaining the portrait lighting image by performing the portraitlighting processing on the target 3D model includes: acquiringsimulation lighting parameters, the simulation lighting parametercomprising at least one of the following: an illumination angle of asimulation light source, a distance between the simulation light sourceand the target 3D model, an amount of the simulation light source, acolor temperature of the simulation light source, or a light intensityof the simulation light source; and obtaining the portrait lightingimage by performing simulation lighting processing on the target 3Dmodel based on the simulation lighting parameters.

In the method according to some embodiments of the present disclosure,obtaining the portrait lighting image by performing the portraitlighting processing on the target 3D model includes: obtaining alight-and-shadow image of the portrait by performing the portraitlighting processing on the target 3D model; and obtaining the portraitlighting image by rendering the image to be processed by using thelight-and-shadow image of the portrait.

In the method according to some embodiments of the present disclosure,obtaining the portrait lighting image by rendering the image to beprocessed using the light-and-shadow image of the portrait includes:obtaining the portrait lighting image by multiplying a color value of apixel in the image to be processed by a color value of a pixel in thelight-and-shadow image of the portrait.

In the method according to some embodiments of the present disclosure,obtaining the light-and-shadow image of the portrait by performing theportrait lighting processing on the target 3D model includes:determining an illumination range of the simulation light source in thetarget 3D model based on a simulation lighting parameter, anddetermining an illumination value of a pixel within the illuminationrange in the target 3D model based on the simulation lighting parameter;and obtaining the light-and-shadow image of the portrait by summing theillumination value and a pixel value corresponding to the illuminationvalue in the target 3D model.

The method according to some embodiments of the present disclosurefurther includes, subsequent to acquiring the simulation lightingparameters: obtaining a detection result by detecting an light intensityof the target portrait in the image to be processed; reducing the lightintensity of the simulation light source in the simulation lightingparameter in the event that the detection result is that the lightintensity is above a first set threshold; and increasing the lightintensity of the simulation light source in the simulation lightingparameter in the event that the detection result is that the lightintensity is below a second set threshold.

In the method according to some embodiments of the present disclosure,obtaining the target 3D model by performing the 3D reconstructionoperation on the target portrait in the image to be processed includes:extracting a key point of the target portrait from the image to beprocessed; and inputting the key point of the target portrait into anartificial neural network to obtain the target 3D model of the targetportrait.

In the method according to some embodiments of the present disclosure,extracting the first image comprising the target portrait from theportrait lighting image includes: separating the target portrait from abackground image in the portrait lighting image by body segmentation toobtain the first image comprising the target portrait from the portraitlighting image.

The method according to some embodiments of the present disclosurefurther includes: acquiring an original image of the image to beprocessed; and acquiring a distortion factor of a photographing device,performing distortion correction on the target portrait in the originalimage by using the distortion factor to obtain the image to beprocessed.

The method according to some embodiments of the present disclosurefurther includes: beautifying the target portrait in the original imageof the image to be processed to obtain the image to be processed.

The method according to some embodiments of the present disclosurefurther includes, subsequent to merging the second image comprising thephotographic background cloth and the first image into the target image:acquiring a material image to be added; and adding the material imageinto the target image, so that an image layer where the material imageis located is above an image layer where the target portrait in thetarget image is located.

At least one embodiment of the present disclosure provides an imageprocessing apparatus includes: an acquisition unit configured to acquirean image to be processed, the image to be processed comprising a targetportrait; a 3D reconstruction unit configured to obtain a target 3Dmodel by performing a 3D reconstruction operation on the target portraitin the image to be processed; a lighting processing unit configured toobtain a portrait lighting image by performing portrait lightingprocessing on the target 3D model, and extract a first image comprisingthe target portrait from the portrait lighting image; and an imagemerging unit configured to merge a second image comprising aphotographic background cloth and the first image into a target image,the second image being used as a background image of the first image.

At least one embodiment of the present disclosure provides an electronicdevice, which includes a processor, a memory storing a computer programbeing capable of being executed by the processor, the computer programwhen executed by the processor, causing the processor to perform themethod according to at least one embodiment of the present disclosure.

At least one embodiment of the present disclosure provides a computerstorage medium, storing a computer program, the computer program whenexecuted by a computer, causing the computer to perform the methodaccording to at least one embodiment of the present disclosure.

Other characteristics and advantages of the present disclosure will beset forth in the following description, partially apparent from thedescription or understood by the implementation of the presentdisclosure. The objectives and other advantages of the presentdisclosure are implemented and achieved in the structures particularlypointed out in the description, the claims and the drawings.

Detailed description will be given below to the exemplary embodimentswith reference to the accompanying drawings to provide a more clearunderstanding of the objectives, the characteristics and the advantagesof the present disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions of theembodiments of the present disclosure, brief description will be givenbelow to the accompanying drawings required to be used in thedescription of the embodiments. Apparently, the drawings described belowonly involve some embodiments of the present disclosure. Other drawingsmay also be obtained by those skilled in the art without creativeefforts on the basis of these drawings.

FIG. 1 is a schematic diagram of an electronic device provided by someembodiments of the present disclosure;

FIG. 2 is a flowchart of an image processing method provided by someembodiments of the present disclosure;

FIG. 3 is a schematic diagram of an optional image to be processedaccording to some embodiments of the present disclosure;

FIG. 4 is a schematic diagram of an optional target 3D model accordingto some embodiments of the present disclosure;

FIG. 5 is a schematic diagram of an optional light-and-shadow image of aportrait according to some embodiments of the present disclosure;

FIG. 6 is a schematic diagram of an optional image to be processed,obtained after being processed, according to some embodiments of thepresent disclosure;

FIG. 7 is a flowchart of an optional image processing method provided bysome embodiments of the present disclosure;

FIG. 8 is a flowchart of another optional image processing methodprovided by some embodiments of the present disclosure; and

FIG. 9 is a schematic diagram of an image processing device provided bysome embodiments of the present disclosure.

DETAILED DESCRIPTION

In order to make objects, technical solutions and advantages of theembodiments of the disclosure apparent, the technical solutions of theembodiments will be described clearly and completely in conjunction withthe drawings related to the embodiments of the disclosure. Apparently,the described embodiments are just a part but not all of the embodimentsof the disclosure. Based on the described embodiments herein, thoseskilled in the art can obtain other embodiment(s), without any inventivework, which should be within the scope of the disclosure.

Firstly, an electronic device 100 provided by some embodiments of thepresent disclosure is described with reference to FIG. 1. The electronicdevice may be used for executing an image processing method provided byvarious embodiments of the present disclosure.

As shown in FIG. 1, the electronic device 100 comprises one or moreprocessors 102, one or more memories 104, an input unit 106, an outputunit 108, a photographing device 110, a radio-frequency (RF) circuit112, an external interface 114 and a communication module 116. Thesecomponents are connected with each other through a bus system 118 and/orconnecting mechanisms in other forms (not shown). It could be understoodthat the components and the structures of the electronic device 100 asshown in FIG. 1 are only illustrative and not limiting. The electronicdevice may also include other components and structures as required.

The processor 102 may be implemented by at least one of the followinghardware: a digital signal processor (DSP), a field programmable gatearray (FPGA), a programmable logic array (PLA) or anapplication-specific integrated circuit (ASIC). The processor 102 may bea central processing unit (CPU) or a processing unit of other formshaving data processing capabilities and/or instruction executioncapabilities, and the processor 102 may control other components in theelectronic device 100 to execute required functions.

The memory 104 may include one or more computer program products. Thecomputer program products may include various forms of computer readablestorage media, e.g., volatile memories and/or nonvolatile memories. Thevolatile memory, for instance, may include a random access memory (RAM)and/or a cache. The nonvolatile memory, for instance, may include aread-only memory (ROM), a hard disk, a flash memory, etc. One or morecomputer program instructions may be stored on the computer readablestorage medium. The processor 102 may execute the program instructions,so as to realize the client function as described below in theembodiment of the present disclosure (implemented by the processor)and/or other required functions. Various applications and various data,e.g., data used and/or produced by the applications, may also be storedin the computer readable storage medium.

The input unit 106 may be a device that is used by a user to inputinstructions, and may include one or more selected from a keyboard, amouse, a microphone and a touch panel.

The output unit 108 may output various information (e.g., image orsound) to the outside (for instance, the user), and may include one ormore of a display, a loudspeaker, etc.

The photographing device 110 is used for image acquisition. A targetimage is obtained by processing data acquired by the photographingdevice by an image processing method. For instance, the photographingdevice may capture images desired by the user (e.g., photos and videos),and the image is processed by the image processing method to obtain atarget image. The photographing device may also store the capturedimages in the memory 104 for the use of other components.

The RF circuit 112 receives and sends electromagnetic waves. The RFcircuit 12 converts electrical signals into electromagnetic waves, orconverts electromagnetic waves into electrical signals, and communicateswith a communication network and other communication equipment throughthe electromagnetic waves. The RF circuit 112 may include known circuitsfor executing the functions, including but not limited to an antennasystem, an RF transceiver, one or more amplifiers, a tuner, one or moreoscillators, a DSP, a CODEC chip set, a subscriber identity module (SIM)card and a memory. The RF circuit 112 may communicate with the networkand other devices through wireless communication. The network includes,for instance, the Internet, also referred to as the World Wide Web(WWW), an intranet and/or a wireless network such as a cellulartelephone network, a wireless local area network (WLAN) and/or ametropolitan area network (MAN). The wireless communication may adoptany of a variety of communication standards, protocols and technologies,including but not limited to Global System for Mobile communications(GSM), Enhanced Data GSM Environment (EDGE), wideband code divisionmultiple access (W-CDMA), code division multiple access (CDMA),time-division multiple address (TDMA), Bluetooth, Wireless Fidelity(Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g and/or IEEE802.11n), Voice over IP (VoIP), Wi-MAX, a protocol for email, instantmessaging and/or Short Message Service (SMS), or any other suitablecommunication protocol, including communication protocols that have notbeen developed as of the filing date of this document.

The communication module 116 may communicate with other devices throughone or more external ports 114, and may include a variety of softwarecomponents for processing data received by the RF circuit 112 and/or theexternal port 114. The external port 114 (e.g., universal serial bus(USB) or FIREWIRE) is applicable to be directly or indirectly coupled toother devices via a network (such as the Internet or WLAN).

Illustratively, the electronic device for implementing the imageprocessing method provided by the embodiments of the present disclosuremay be implemented as a smart mobile terminal such as a smart mobilephone or a tablet PC.

Some embodiments of the present disclosure provides an image processingmethod. It should be noted that the steps shown in the flowchart of theaccompanying drawings may be executed in a computer system, such as aset of computer-executable instructions. Moreover, although a logicalorder is shown in the flowchart, in some cases, the steps shown ordescribed may be executed in an order different from the one describedherein.

FIG. 2 is a flowchart of an image processing method provided by someembodiments of the present disclosure. As shown in FIG. 2, the methodcomprises the following steps.

S202: acquiring an image to be processed, the image to be processedincluding a target portrait. The target portrait may be a full orpartial facial image, may be a full-body image of a person, and may alsobe a half-length image of a person, which is not specifically limited inthe embodiment.

S204: obtaining a target 3D model by performing a 3D reconstructionoperation on the target portrait in the image to be processed.

S206: obtaining a portrait lighting image by performing portraitlighting processing on the target 3D model, and extracting a first imageincluding the target portrait from the portrait lighting image. Theportrait lighting processing in the embodiment is to process the target3D model by simulating the lighting manner of professional studios, sothat the image to be processed after processing can restore the studiolighting method in the physical world as much as possible.

S208: merging a second image including a photographic background clothand the first image into a target image, the second image being used asa background image of the first image.

It should be noted that in the embodiment, the steps S202 to S208 may beapplied on smart mobile terminals such as smart mobile phones and tabletPCs.

When the method is applied on the smart mobile terminal, a virtualstudio-like portrait effect can be produced by photographs taken by theportrait photographing function of the smart mobile terminal. When theuser wishes to take a formal photo, the user does not need to find aprofessional studio. Particularly when the user urgently needs a formalphoto, it can be realized easily and quickly by the smart mobileterminal.

In the embodiments of the present disclosure, the target 3D model isobtained by performing the 3D reconstruction operation on the targetportrait in the image to be processed; the portrait lighting image isobtained by performing the portrait lighting processing on the target 3Dmodel, and the first image including the target portrait is extractedfrom the portrait lighting image, and after the image to be processed isprocessed by the above processes, the studio lighting manner in thephysical world can be restored as much as possible; and the target imageis obtained by merging the second image including the photographicbackground cloth and the first image. In the embodiments of the presentdisclosure, the above-mentioned processing method may be adopted tosolve the technical problem that the virtual studio-like portrait effectcannot be produced by the photographs taken by the portraitphotographing function of the conventional smart mobile terminal, sothat the photographs taken by the portrait photographing function of thesmart mobile terminal may produce the virtual studio-like portraiteffect.

In the embodiments of the present disclosure, the steps S202 to S208 maybe performed on a large number of images; or the steps S202 to S208 mayalso be executed after one image is acquired each time. No specificlimitation will be given in the embodiment. The image processing methodwill be described below with reference to the specific embodiments.

As known from the above description, in the embodiments, an image to beprocessed which include a target portrait is acquired at first. Theimage to be processed may be an original image that is captured by thephotographing device of the smart mobile terminal and is not processed.The image to be processed may also be an image that is captured by thephotographing device of the smart mobile terminal and is processed. Morespecifically, the processing may be distortion correction and/orbeautifying.

In an optional embodiment, an original image of the image to beprocessed, captured by the photographing device of the smart mobileterminal, is acquired; and then, the image to be processed is obtainedby acquiring a distortion factor of the photographing device andutilizing the distortion factor to perform distortion correction on thetarget portrait in the original image.

More specifically, as the photographing device on the smart mobileterminal is usually a wide-angle lens, the smart mobile terminal willproduce image distortion when taking self-portraits. Distortion belongsto the geometric distortion of images, and is a picture distortionphenomenon caused by different magnifications in different regions onthe focal plane. However, studios in the physical world use a portraitlens having a focal length of 55 mm or more, which does not cause imagedistortion. In most cases, these distortions need to be corrected to theextent that normal human eyes do not see any distortion. Therefore, adistortion correction algorithm is required to solve this problem.

In the embodiments, the image to be processed is obtained by adoption ofdistortion correction algorithm to perform distortion correction on thetarget portrait in the image to be processed.

Specifically, parameter calibration may be performed in advance to thephotographing device of the smart mobile terminal to determine thedistortion factor of the photographing device; and after the distortionfactor of the photographing device is determined, when taking a photo,the photographing device can perform distortion correction on the photoby the distortion correction algorithm and then output the photo.

It should be noted that in the embodiment, in the process of performingdistortion correction on the original image, distortion correction maybe only performed on the target portrait in the original image.Specifically, key points of the target portrait may be detected in theoriginal image; after the key points are obtained, an area where theportrait is located is determined based on the key points; andsubsequently, the target portrait is subjected to distortion correction.More specifically, pixels of the target portrait may be subjected todistortion correction by utilization of the distortion factor, and theimage to be processed is obtained after the distortion correction.

In another optional embodiment, an original image (the original image ofthe image to be processed) captured by the photographing device of thesmart mobile terminal is acquired; and then, the image to be processedis obtained by beautifying the target portrait in the original image ofthe image to be processed.

In the embodiment, a beautifying algorithm is utilized to solve theproblem of facial makeup of the target portrait and prepare forstudio-portrait photography.

Optionally, the steps of implementing the beautifying algorithminclude: 1. blurring the image by a filtering algorithm with anedge-preserving effect; 2. adopting a skin color detection algorithm toprotect non-skin areas; 3. merging the blurred image and the originalimage; and 4. sharpening the merged image.

It should be noted that apart from the above-mentioned beautifyingfunctions, other beautifying functions may also be set, e.g., whitening,definition and screenshot. No specific limitation is given in theembodiments.

Moreover, it should be noted that in the embodiment, before taking aphoto by the smart mobile terminal according to the steps as describedabove, beautifying parameters such as whiteness, definition, andskin-smoothing degree may be manually set in advance. After the settingsare completed, in the process of beautifying the original image, theoriginal image may be beautified according to the set beautifyingparameters.

In another optional embodiment, after acquiring the original image (theoriginal image of the image to be processed) captured by thephotographing device of the smart mobile terminal, the original imagemay be sequentially processed in combination with the above-mentioneddistortion correction algorithm and the beautifying algorithm. Thespecific processing sequence is not specifically limited in theembodiment.

In the embodiment, after the image to be processed is obtained by theabove-mentioned processing method, the target 3D model may be obtainedby performing a 3D reconstruction operation on the target portrait inthe image to be processed. After the target 3D model is obtained, theportrait lighting image is obtained by performing portrait lightingprocessing on the target 3D model.

As the professional studios use different lighting methods to brightenthe face and the body, but the hardware of the mobile phones does nothave the above lighting abilities, in the embodiment, the portraitlighting method is adopted to process the target portrait, so as tosimulate the lighting manner of the professional studio, and then theobtained portrait lighting image can restore the studio lighting effectin the physical world as much as possible.

In the embodiment, the objective is to simulate and restore the lightingeffect of professional studios in the physical world as much aspossible, so that it looks like a professional lighter is lighting whenconsumers take pictures. Specifically, the above objective may beachieved by the following method: generating 3D information on thetarget portrait to obtain a 3D model (namely the above target 3D model)of the target portrait; and freely controlling simulation lightingparameters in a render layer, and performing simulation lightingprocessing on the 3D model of the target portrait based on thesimulation lighting parameters. The process will be described below indetail.

In an optional embodiment, the step S204 of obtaining the target 3Dmodel by performing the 3D reconstruction operation on the targetportrait in the image to be processed includes: extracting key points ofthe target portrait from the image to be processed; and inputting thekey points of the target portrait into an artificial neural network forprocessing to obtain the target 3D model of the target portrait.

In the embodiment, the key points (for instance, key points of the humanface) of the target portrait in the image to be processed are extracted;and after the key points of the target portrait are extracted, the keypoints of the target portrait may be processed via the artificial neuralnetwork, so as to obtain the target 3D model of the target portrait.FIG. 3 shows the image to be processed, and FIG. 4 shows the target 3Dmodel of the human face in the image to be processed.

It should be noted that in the embodiment, the artificial neural networkis a neural network that has been trained beforehand. In the process oftraining the artificial neural network, the inputs are key points of aportrait in a training sample, and the outputs is a 3D model of thetraining sample.

After the target 3D model is obtained, the portrait lighting image maybe obtained by performing the portrait lighting processing on the target3D model.

In an optional embodiment, the step S206 of obtaining the portraitlighting image by performing the portrait lighting processing on thetarget 3D model includes the following steps.

S11: obtaining a light-and-shadow image of the portrait by performingthe portrait lighting processing on the target 3D model.

S12: obtaining the portrait lighting image by rendering the image to beprocessed by using the light-and-shadow image of the portrait.

Optionally, the step S12 of obtaining the portrait lighting image byrendering the image to be processed by using the light-and-shadow imageof the portrait includes: obtaining the portrait lighting image bymultiplying the color value of a pixel in the image to be processed bythe color value of a pixel in the light-and-shadow image of theportrait.

In the embodiment, during performing the portrait lighting processing onthe target 3D model, the target 3D model may be subjected to theportrait lighting processing based on simulation lighting parameters. Itis well known that in the process of shooting in a professional studio,the photographer will place one or more photographic lamps around asubject to be photographed. In the embodiment, in order to restore thelighting manner of the professional studio in the physical world as muchas possible, the target 3D model is subjected to the simulation lightingprocessing by setting the simulation lighting parameters, and thelighting effect in the processed image may simulate the lighting effectof the professional studio in the physical world.

More specifically, in the embodiment, the simulation lighting parametersinclude at least one of the following: the illumination angle of asimulation light source, the distance between the simulation lightsource and the target 3D model, the amount of the simulation lightsource, the color temperature of the simulation light source, or thelight intensity of the simulation light source. For instance, a whitesimulation point light source is set, and the light source is set toilluminate from the left side of the target portrait at 70 degrees.After the simulation lighting parameters are set, simulation lightingprocessing may be performed on the target 3D model based on thesimulation lighting parameters, and for instance, a light-and-shadowimage of the portrait as shown in FIG. 5 may be obtained.

After the light-and-shadow image of the portrait is obtained, the imageto be processed may be rendered by using the light-and-shadow image ofthe portrait to obtain the portrait lighting image. Supposing the imageto be processed is as shown in FIG. 3, after the image is processed bythe above-mentioned method, the target 3D model as shown in FIG. 4 isobtained, and the light-and-shadow image of the portrait as shown inFIG. 5 is obtained. Subsequently, an image as shown in FIG. 6 (namelythe portrait lighting image) may be obtained by rendering the image tobe processed as shown in FIG. 3 by using the light-and-shadow image ofthe portrait as shown in FIG. 5.

In the embodiments, one implementation of rendering the image to beprocessed by using the light-and-shadow image of the portrait is tomultiply the color values of pixels in the image to be processed by thecolor values of pixels in the light-and-shadow image of the portrait, soas to render the image to be processed by using the light-and-shadowimage of the portrait.

In another optional embodiment, the step S206 of obtaining the portraitlighting image by performing the portrait lighting processing on thetarget 3D model includes the following steps.

S21: acquiring simulation lighting parameters, the simulation lightingparameter includes at least one of the following: the illumination angleof the simulation light source, the distance between the simulationlight source and the target 3D model, the amount of the simulation lightsources, the color temperature of the simulation light source, or thelight intensity of the simulation light source.

S22: obtaining the portrait lighting image by performing simulationlighting processing on the target 3D model based on the simulationlighting parameters.

In the embodiments, the simulation lighting parameters are acquired, andthen the portrait lighting image is obtained by performing thesimulation lighting processing on the target 3D model based on thesimulation lighting parameters. More specifically, the process includes:determining the illumination range of the simulation light source in thetarget 3D model based on the simulation lighting parameter, anddetermining the illumination value of pixels within the illuminationrange in the target 3D model based on the simulation lighting parameter;obtaining the light-and-shadow image of the portrait by summing theillumination value and a pixel value corresponding to the illuminationvalue in the target 3D model; and obtaining the portrait lighting imageby rendering the image to be processed by using the light-and-shadowimage of the portrait.

In the embodiments, in order to restore the lighting manner of theprofessional studio in the physical world as much as possible, thetarget 3D model is subjected to the simulation lighting processing bysetting the simulation lighting parameters, and the lighting effect inthe processed image may simulate the lighting effect of the professionalstudio in the physical world.

In another optional embodiment, after the simulation lighting parametersare acquired, the method further comprises the following steps:obtaining a detection result by detecting the light intensity of thetarget portrait in the image to be processed; reducing the lightintensity of the simulation light sources in the simulation lightingparameters in the event that the detection result is that the lightintensity is above a first set threshold; and increasing the lightintensity of the simulation light sources in the simulation lightingparameters in the event that the detection result is that the lightintensity is below a second set threshold.

More specifically, in the embodiment, the simulation lighting parametersare set in advance, and the simulation lighting parameters include thefollowing types of parameters: the illumination angle of the simulationlight source, the distance between the simulation light source and thetarget 3D model, the amount of the simulation light source, the colortemperature of the simulation light source, and the light intensity ofthe simulation light source. Before performing the portrait lightingprocessing on the target 3D model, the light intensity of the targetportrait in the image to be processed may be detected to obtain thedetection result.

In the event that it is determined that the light intensity of thetarget portrait in the image to be processed is above the first setthreshold based on the detection result, the light intensity of thesimulation light source in the simulation lighting parameters is reducedby, for instance, multiplying the light intensity of the simulationlight source in the simulation lighting parameters by a percentage lessthan 1. In the event that it is determined that the light intensity ofthe target portrait in the image to be processed is below the second setthreshold based on the detection result, the light intensity of thesimulation light source in the simulation lighting parameters isincreased by, for instance, multiplying the light intensity of thesimulation light source in the simulation lighting parameters by apercentage greater than 1.

After the processes as described above, the acquired simulation lightingparameters may be adjusted based on the light intensity of the targetportrait in the image to be processed. It should be noted that in theembodiment, one objective of processing by the above-mentioned method isto prevent the light intensity of the target portrait in the portraitlighting image from being too high, or prevent the light intensity ofthe target portrait in the portrait lighting image from being not highenough, as the light intensity which is too high or not high enough willaffect the display effect of the portrait lighting image.

In another optional embodiment, the step of obtaining thelight-and-shadow image of the portrait by the portrait lighting of thetarget 3D model includes the following steps: determining theillumination range of the simulation light source in the target 3D modelbased on the simulation lighting parameter, and determining theillumination value of pixels within the illumination range in the target3D model based on the simulation lighting parameter; and obtaining thelight-and-shadow image of the portrait by summing the illumination valueand a pixel value corresponding to the illumination value in the target3D model.

In the embodiments, the position coordinate of the simulation lightsource in the target 3D model may be determined based on the determinedsimulation lighting parameters. For instance, the determined positioncoordinate is (x, y). After the position coordinate is determined, theillumination range of the simulation light source may be determinedbased on the position coordinate. Subsequently, the illumination valueof pixels within the illumination range may be determined. Based on thecharacteristics of the light source, the light near the light source isstrong, and the light away from the light source is weak and shadowswill be produced. Therefore, in the embodiments, the illumination valueof the pixels within the illumination range may be determined based onthe characteristics of the light source.

After the illumination value is obtained, the illumination value and thepixel value corresponding to the illumination value in the target 3Dmodel are summed, and the light-and-shadow image of the portrait may beobtained after summing. Corresponding pixel values in the target 3Dmodel refer to the pixel value of the pixel within the illuminationrange in the target 3D model.

In the embodiments, after the image to be processed is processed by theabove-mentioned method, the first image including the target portraitmay be extracted from the portrait lighting image.

In an optional embodiment, the step of extracting the first imageincluding the target portrait from the portrait lighting image includesthe following steps: separating the target portrait and a backgroundimage in the portrait lighting image by body segmentation, and obtainingthe first image including the target portrait from the portrait lightingimage. After the first image is obtained, a second image including aphotographic background cloth and the first image may be merged into atarget image.

Since the shooting scenarios of the smart mobile terminal areuncontrollable, the shooting background is often messy. The professionalstudio is in a closed space with artificial scenery, so the shootingbackground is often simple. Therefore, in the embodiments, the body ofthe portrait (the image to be processed after processing) should beseparated from the background by body segmentation, and the studiobackground cloth is converted into an RGB image (namely the secondimage) which is bonded to a background layer of the first image, so asto realize the effect of virtual green screen/virtual background.

It should be noted that in the embodiments, the second image in theimage library has multiple types, and before the image to be processedis processed by the above-mentioned method, the user may select onesecond image as a background image according to actual demands. If theuser selects the background image in advance, the target image will besynthesized based on the default background image.

In another optional embodiment, the step S108 of merging the secondimage including the photographic background cloth and the first imageinto the target image includes the following steps.

S1081: taking the second image as an image of the background layer, andtaking the first image as an image of a foreground layer.

S1082: determining pixels, corresponding to pixels of the image of theforeground layer, in the image of the background layer based on positioninformation on the first image in the image to be processed.

S1083: replacing the pixels of the image of the background layer withthe pixels of the image of the foreground layer corresponding to thepixels of the image of the background layer to obtain the target image.

In the embodiments, the second image including the photographicbackground cloth is taken as the image of the background layer, thefirst image is taken as the image of the foreground layer, and then theimage of the background layer and the image of the foreground layer aremerged.

In the process of merging the image of the background layer and theimage of the foreground layer, as the size of the image of thebackground layer is the same with the size of the image to be processed,in the embodiments, the position information of the target portrait (thefirst image) in the image to be processed may be determined, and theposition of the first image is determined in the image of the backgroundlayer based on the position information; then, pixels corresponding tothe first image are determined at the position; and finally, thedetermined pixels are replaced with the pixels of the first image, so asto merge the image of the background layer and the image of theforeground layer.

In the embodiments, after the second image including the photographicbackground cloth and the first image are merged into the target image,an material image to be added may also be acquired; and the materialimage is added into the target image, so that an image layer where thematerial image is located is disposed above an image layer where thetarget portrait in the target image is located. The material imagesrefer to materials in the form of image, and the material images showelements that may be used to form an image.

Specifically, in the embodiment, problems in retouching may be solved byutilization of material overlay and filters, making the scene of theportrait more enriched and more colorful.

It should be noted that the material image to be added may be a defaultmaterial image and may also be a material image selected by the user inadvance.

For instance, a material image library includes a variety of materialimages, and the user may select at least one material image in each kindof material images as the material image to be added.

As can be seen from the above description, in the embodiments of thepresent disclosure, the above-mentioned processing method can be adoptedto solve the technical problem that the portrait photography using theexisting smart terminal devices cannot produce a virtual studio-likeportrait effect, so that the portrait photography using the smartterminal device can produce the virtual studio-like portrait effect.

FIG. 7 is a flowchart of an optional image processing method provided bysome embodiments of the present disclosure. As shown in FIG. 7, themethod comprises the following steps.

S701: acquiring an original image of an image to be processed.

S702: acquiring a distortion factor of a photographing device,performing distortion correction on a target portrait in the originalimage based on the distortion factor to obtain a corrected image.

S703: obtaining the image to be processed by beautifying the targetportrait in the corrected image. In the embodiments, a beautifyingalgorithm is utilized to solve the problem of facial makeup of thetarget portrait and prepare for studio portrait photography.

S704: obtaining a target 3D model by performing a 3D reconstructionoperation on the target portrait in the image to be processed via aartificial neural network. More specifically, step S704 includes:extracting key points of the target portrait from the image to beprocessed; and inputting the key points of the target portrait into theartificial neural network to obtain the target 3D model of the targetportrait.

S705: obtaining a light-and-shadow image of the portrait by performingthe portrait lighting processing on the target 3D model.

S706: obtaining a portrait lighting image by rendering the image to beprocessed by using the light-and-shadow image of the portrait.

S707: separating the target portrait and a background image in theportrait lighting image by body segmentation, and obtaining a firstimage including the target portrait from the portrait lighting image.

S708: merging a second image including a photographic background clothand the first image into a target image, the second image in the targetimage being a background image of the first image.

S709: acquiring a material image to be added.

S710: adding the material image into the target image, so that an imagelayer where the material image is located is disposed above an imagelayer where the target portrait in the target image is located.

The implementation of the steps S701 to S710 is as described above, andthe detailed description of which will not be repeated herein.

FIG. 8 is a flowchart of another optional image processing methodprovided by the embodiment of the present disclosure. As shown in FIG.8, the method comprises the following steps.

S801: acquiring an image to be processed, which includes a targetportrait.

S802: extracting key points from the image to be processed.

S803: obtaining a target face model by performing a 3D facereconstruction operation on the target portrait in the image to beprocessed based on the key points.

Optionally, the step of obtaining the target face model by performingthe 3D face reconstruction operation on the target portrait in the imageto be processed based on the key points includes: obtaining the target3D model by performing the 3D reconstruction operation on the targetportrait in the image to be processed via a artificial neural network.

S804: obtaining a light-and-shadow image of the portrait by performingthe portrait lighting processing on the target 3D model.

S805: obtaining a portrait lighting image by rendering the image to beprocessed by using the light-and-shadow image of the portrait.

S806: extracting a first image including the target portrait from theportrait lighting image.

S807: merging a second image including a photographic background clothand the first image into a target image. At step S807, the second imagein the target image is a background image of the first image.

The implementation of the steps S801 to S807 is as described in theabove embodiments, and detailed description of which will not berepeated.

Some embodiments of the present disclosure further provides an imageprocessing apparatus. The image processing apparatus is mainly used forexecuting the foregoing image processing method provided by theembodiments of the present disclosure. Detailed description will begiven below to the image processing method provided by the embodiment ofthe present disclosure.

FIG. 9 is a schematic diagram of an image processing apparatus providedby the embodiment of the present disclosure. As shown in FIG. 9, theimage processing apparatus mainly comprises an acquisition unit 10, a 3Dreconstruction unit 20, a lighting processing unit 30 and an imagemerging unit 40.

The acquisition unit 10 is configured to acquire an image to beprocessed, which includes a target portrait.

The 3D reconstruction unit 20 is configured to obtain a target 3D modelby performing a 3D reconstruction operation on the target portrait inthe image to be processed.

The lighting processing unit 30 is configured to obtain a portraitlighting image by performing portrait lighting processing on the target3D model, and extract a first image including the target portrait fromthe portrait lighting image.

The image merging unit 40 is configured to merge a second imageincluding a photographic background cloth and the first image into atarget image, and the second image is a background image of the firstimage.

In the embodiments of the present disclosure, the target 3D model isobtained by performing the 3D reconstruction operation on the targetportrait in the image to be processed; the portrait lighting image isobtained by performing the portrait lighting processing on the target 3Dmodel, and the first image including the target portrait is extractedfrom the portrait lighting image, and after the image to be processed isprocessed by the above processes, the studio lighting manner in thephysical world can be restored as much as possible; and the second imageincluding the photographic background cloth and the first image aremerged into the target image. In the embodiments of the presentdisclosure, the above-mentioned processing method can be adopted tosolve the technical problem that the virtual studio-like portrait effectcannot be produced by the photographs taken by the portraitphotographing function of the conventional smart mobile terminal, sothat the virtual studio-like portrait effect can be produced by thephotographs taken by the portrait photographing function of the smartmobile terminal.

Optionally, the image merging unit is configured to: take the secondimage as an image of a background layer, and take the first image as animage of a foreground layer; determine pixels, corresponding to pixelsof the image of the foreground layer, in the image of the backgroundlayer based on position information on the first image in the image tobe processed; and replace corresponding pixels with the pixels of theimage of the foreground layer to obtain the target image.

Optionally, the lighting processing unit 30 includes: an acquisitionmodule configured to acquire simulation lighting parameters, whichinclude at least one of the following: the illumination angle of thesimulation light source, the distance between the simulation lightsource and the target 3D model, the amount of the simulation lightsource, the color temperature of the simulation light source, or thelight intensity of the simulation light source; and an simulationlighting module configured to obtain the portrait lighting image byperforming simulation lighting processing on the target 3D model basedon the simulation lighting parameters.

Optionally, the lighting processing unit 30 may further include: aportrait lighting processing module configured to obtain alight-and-shadow image of the portrait by performing portrait lightingprocessing on the target 3D model; and a rendering module configured torender the image to be processed by using the light-and-shadow image ofthe portrait to obtain the portrait lighting image.

Optionally, the rendering module is configured to: obtain the portraitlighting image by multiplying the color values of pixels in the image tobe processed by the color values of pixels in the light-and-shadow imageof the portrait.

Optionally, the portrait lighting module is configured to: determine theillumination range of the simulation light source in the target 3D modelbased on the simulation lighting parameter, and determine theillumination value of pixels within the illumination range in the target3D model based on the simulation lighting parameter; and obtain thelight-and-shadow image of the portrait by summing the illumination valueand the pixel value corresponding to the illumination value in thetarget 3D model.

Optionally, the apparatus is also configured to: obtain a detectionresult by detecting the light intensity of the target portrait in theimage to be processed after acquiring the simulation lightingparameters; reduce the light intensity of the simulation light source ininitial simulation lighting parameters and obtain the simulationlighting parameters, in the event that the detection result is that thelight intensity is above a first set threshold; and increase the lightintensity of the simulation light source in the initial simulationlighting parameters and obtain the simulation lighting parameters, inthe event that the detection result is that the light intensity is belowa second set threshold.

Optionally, the 3D reconstruction unit 20 is configured to: extract keypoints of the target portrait from the image to be processed; and inputthe key points of the target portrait into the artificial neural networkto obtain the target 3D model of the target portrait.

Optionally, the lighting processing unit further includes: a separationmodule configured to separate the target portrait and a background imagein the portrait lighting image by body segmentation, and obtain thefirst image including the target portrait from the portrait lightingimage.

Optionally, the apparatus is also configured to: acquire an originalimage of the image to be processed; and acquire a distortion factor of aphotographing device, perform distortion correction on the targetportrait in the original image based on the distortion factor to obtainthe image to be processed.

Optionally, the apparatus is also configured to acquire a material imageto be added after merging the second image including the photographicbackground cloth and the first image into the target image; and add thematerial image into the target image, so that the image layer where thematerial image is located is disposed above the image layer where thetarget portrait in the target image is located.

The implementation principles and the technical effects of the apparatusprovided by the embodiments of the present disclosure are the same asthose of the foregoing method embodiments. For brief description, thosenot mentioned in the device embodiment may refer to the correspondingcontent in the foregoing method embodiments.

In addition, in the description of the embodiments of the presentdisclosure, unless explicitly stated and defined otherwise, the terms“mounted”, “connected with”, and “connected to” are to be understoodbroadly, for instance, may be fixed connection, detachable connection orintegral connection; may be mechanical connection or electricalconnection; may be directly connected, may be indirectly connectedthrough an intermediate medium, or may be internal communication betweentwo elements. The specific meanings of the above terms in the presentdisclosure can be understood by those skilled in the art according tospecific conditions.

In the description of the present disclosure, it should be noted thatthe orientation or positional relationship indicated by the terms“center”, “on”, “beneath”, “left”, “right”, “vertical”, “horizontal”,“inside”, “outside” or the like is based on the orientation orpositional relationship shown in the accompanying drawings, is merelyfor the convenience of describing the present disclosure and thesimplified description, and does not indicate or imply that the deviceor component referred to has a specific orientation and is constructedand operated in a specific orientation, and therefore shall not beconstrued as the limitation to the present disclosure. In addition, theterms “first”, “second”, and “third” are used for descriptive purposesonly and shall not be construed as indicating or implying the relativeimportance.

It could be clearly understood by those skilled in the art that for theconvenience and brevity of the description, the specific workingprocesses of the system, the device and the unit described above mayrefer to corresponding processes in the foregoing method embodiments,and detailed description will not be repeated herein.

In the several embodiments provided by the present application, itshould be understood that the disclosed system, device and method may beimplemented in other manners. The device embodiments described above aremerely illustrative. For example, the division of the unit is only alogical function division. In actual implementations, there may be otherdivision manners. Moreover, for example, multiple units or componentsmay be combined or may be integrated into another system, or somecharacteristics may be ignored or not executed. In addition, the mutualcoupling or direct coupling or communication connection shown ordiscussed may be indirect coupling or communication connection throughsome communication interfaces, devices or units, and may be electrical,mechanical or in other forms.

The units described as separate components may or may not be physicallyseparated, and the components displayed as units may or may not bephysical units, that is, may be arranged in one place, or may bedistributed on multiple network units. Some or all of the units may beselected according to actual requirements to achieve the objectives ofthe solutions of the embodiments.

In addition, the functional units in the embodiments of the presentdisclosure may be integrated into one processing unit, or each unit mayexist physically separately, or two or more units may be integrated intoone unit.

The functions, if implemented in the form of software functional unitsand sold or used as separate products, may be stored in aprocessor-executable nonvolatile computer readable storage medium. Basedon such understanding, the technical proposal of the present disclosurein essence, or a portion that contributes to the prior art, or a portionof the technical proposal may be embodied in the form of a softwareproduct. The computer software product is stored in a storage medium,including a plurality of instructions that are used to cause a computerdevice (may be a personal computer, server, or network device) toexecute all or part of the steps of the methods in the embodiments ofthe present disclosure. The foregoing storage medium includes: a USBflash disk (UFD), a mobile hard disk, an ROM, an RAM, a magnetic disk,an optical disk or other media capable of storing program codes.

It should be finally noted that: the foregoing embodiments are merelythe exemplary embodiments of the present disclosure, are used to explainthe technical solutions of the present disclosure, and are not used tolimit the present disclosure. The scope of protection of the presentdisclosure is not limited thereto. Although detailed description hasbeen given to the present disclosure with reference to the foregoingembodiments, it shall be understood by those skilled in the art thatmodification may be made by those skilled in the art to the technicalsolutions described in the foregoing embodiments within the technicalscope disclosed by the present disclosure, or variations may be easilymade, or equivalent replacements may be made to partial technicalfeatures; and these modifications, variations or replacements of thepresent disclosure are not intended to allow the essence ofcorresponding technical solutions to depart from the spirit and scope ofthe technical solutions of the embodiments of the present disclosure,and shall fall within the protection scope of the present disclosure.Therefore, the protection scope of the present disclosure should bedefined by the protection scope of the claims.

What is claimed is:
 1. An image processing method, comprising: acquiringan image to be processed, the image to be processed comprising a targetportrait; obtaining a target three-dimensional (3D) model by performinga 3D reconstruction operation on the target portrait in the image to beprocessed; obtaining a portrait lighting image by performing portraitlighting processing on the target 3D model, and extracting a first imagecomprising the target portrait from the portrait lighting image; merginga second image comprising an image of a photographic background clothand the first image to obtain a target image, the second image beingused as a background image of the first image: acquiring a materialimage to be added; and adding the material image into the target image,so that an image layer where the material image is located is above animage layer where the target portrait in the target image is located. 2.The method according to claim 1, wherein merging the second imagecomprising the image of the photographic background cloth and the firstimage to obtain the target image comprises: taking the second image asan image of a background layer, and taking the first image as an imageof a foreground layer; determining a pixel in the image of thebackground layer corresponding to a pixel of the image of the foregroundlayer, based on position information on the first image in the image tobe processed; and replacing the pixel of the image of the backgroundlayer with the pixel of the image of the foreground layer correspondingto the pixel of the image of the background layer to obtain the targetimage.
 3. The method according to claim 1, wherein obtaining theportrait lighting image by performing the portrait lighting processingon the target 3D model comprises: acquiring simulation lightingparameters, the simulation lighting parameter comprising at least one of: an illumination angle of a simulation light source, a distance betweenthe simulation light source and the target 3D model, an amount of thesimulation light source, a color temperature of the simulation lightsource, or a light intensity of the simulation light source; andobtaining the portrait lighting image by performing simulation lightingprocessing on the target 3D model based on the simulation lightingparameters.
 4. The method according to claim 3, subsequent to acquiringthe simulation lighting parameters, further comprising: obtaining adetection result by detecting a light intensity of the target portraitin the image to be processed; reducing the light intensity of thesimulation light source in the simulation lighting parameter in theevent that the detection result is that the light intensity of thetarget portrait is above a first set threshold; and increasing the lightintensity of the simulation light source in the simulation lightingparameter in the event that the detection result is that the lightintensity of the target portrait is below a second set threshold.
 5. Themethod according to claim 1, wherein obtaining the portrait lightingimage by performing the portrait lighting processing on the target 3Dmodel comprises: obtaining a light-and-shadow image of the targetportrait by performing the portrait lighting processing on the target 3Dmodel; and obtaining the portrait lighting image by rendering the imageto be processed by using the light-and-shadow image of the targetportrait.
 6. The method according to claim 5, wherein obtaining theportrait lighting image by rendering the image to be processed using thelight-and-shadow image of the target portrait comprises: obtaining theportrait lighting image by multiplying a color value of a pixel in theimage to be processed by a color value of a pixel in thelight-and-shadow image of the target portrait.
 7. The method accordingto claim 5, wherein obtaining the light-and-shadow image of the targetportrait by performing the portrait lighting processing on the target 3Dmodel comprises: determining an illumination range of a simulation lightsource in the target 3D model based on a simulation lighting parameter,and determining an illumination value of a pixel within the illuminationrange in the target 3D model based on the simulation lighting parameter;and obtaining the light-and-shadow image of the target portrait bysumming the illumination value and a pixel value corresponding to theillumination value in the target 3D model.
 8. The method according toclaim 1, wherein obtaining the target 3D model by performing the 3Dreconstruction operation on the target portrait in the image to beprocessed comprises: extracting a key point of the target portrait fromthe image to be processed; and inputting the key point of the targetportrait into an artificial neural network to obtain the target 3D modelof the target portrait.
 9. The method according to claim 1, whereinextracting the first image comprising the target portrait from theportrait lighting image comprises: separating the target portrait from abackground image in the portrait lighting image by body segmentation toobtain the first image comprising the target portrait from the portraitlighting image.
 10. The method according to claim 1, further comprising:acquiring an original image of the image to be processed; and acquiringa distortion factor of a photographing device, and performing distortioncorrection on the target portrait in the original image by using thedistortion factor to obtain the image to be processed.
 11. An electronicdevice, comprising a processor, a memory storing a computer programbeing capable of being executed by the processor, the computer programwhen executed by the processor, causing the processor to perform themethod according to claim
 1. 12. A non-transitory computer-readablerecording medium, storing a computer program, the computer program whenexecuted by a computer, causing the computer to perform the methodaccording to claim
 1. 13. An image processing apparatus, comprising: anacquisition unit configured to acquire an image to be processed, theimage to be processed comprising a target portrait; a 3D reconstructionunit configured to obtain a target 3D model by performing a 3Dreconstruction operation on the target portrait in the image to beprocessed; a lighting processing unit configured to obtain a portraitlighting image by performing portrait lighting processing on the target3D model, and extract a first image comprising the target portrait fromthe portrait lighting image; an image merging unit configured to merge asecond image comprising an image of a photographic background cloth andthe first image to obtain a target image, the second image being used asa background image of the first image; a material image acquisition unitconfigured to acquire a material image to be added; and a material imageadding unit configured to add the material image into the target image,so that an image layer where the material image is located is above animage layer where the target portrait in the target image is located.